With the development of the application of computer vision in the field of marine fisheries, fish image retrieval has played a huge role in fishery resource survey and fish behavior analysis. It is found that the background information of fish images can greatly interfere with fish image retrieval, and the fish image retrieval results only using color, texture, shape and other characteristics of fish images are not accurate due to the lack of spatial position information. To solve the above problems, a novel fish image retrieval algorithm based on HSVG (Hue, Saturation, Value, Gray) four-channel and spatial pyramid was proposed. Firstly, a visual saliency map was extracted to separate the foreground and the background, thereby reducing the interference of the image background on the retrieval. Then, in order to contain certain spatial position information, the fish image was converted into an HSVG four-channel map, and on this basis, the theory of spatial pyramid was used to segment the image and extract the SURF (Speed Up Robust Feature). Finally, the search results were obtained. In order to verify the effectiveness of the proposed algorithm, the recall and precision of the algorithm were compared with classic HSVG algorithm and saliency block algorithm on QUT_fish_data dataset and DLOU_fish_data dataset. Compared with traditional HSVG algorithm, the precision on two datasets is increased at most by 12% and 5%, and the recall is increased at most by 7% and 22%, respectively. Compared with saliency block algorithm, the precision on two datasets is increased at most by 15% and 5%, and the recall is increased at most by 36% and 22%, respectively. So, the proposed algorithm is effective and can improve the retrieval results significantly.